Skip to content

docs(methodology): add Bertrand-Duflo-Mullainathan (2004) paper review#553

Merged
igerber merged 1 commit into
mainfrom
docs/placebo-bdm-2004-review
Jun 26, 2026
Merged

docs(methodology): add Bertrand-Duflo-Mullainathan (2004) paper review#553
igerber merged 1 commit into
mainfrom
docs/placebo-bdm-2004-review

Conversation

@igerber

@igerber igerber commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add a paper-sourced methodology review of Bertrand, Duflo & Mullainathan (2004), How Much Should We Trust Differences-in-Differences Estimates?, QJE 119(1):249-275 — docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md.
  • Grounds the PlaceboTests diagnostic in BDM's placebo-law / randomization-inference framework: the two-step random assignment (uniform fake date 1985-1995; 25 of 50 random treated groups), the serial-correlation over-rejection evidence (Table II: 67.5% uncorrected, 44% state-year cluster, 5% serially-uncorrelated), and the four inference corrections BDM evaluate (parametric AR, block/percentile-t bootstrap, state-level cluster VCV, time-series aggregation).
  • Captures the two load-bearing footnotes for the permutation diagnostic: fn 11 (randomization-inference link) and fn 12 (the placebo distribution is not standard normal, so the empirical distribution is the correct reference).

This is the paper-review half (PR-A) of the PlaceboTests methodology promotion. Integrating the proposed ## PlaceboTests registry entry into REGISTRY.md and promoting the METHODOLOGY_REVIEW.md tracker row from In Progress to Complete are the work of the follow-up source-validation pass; the new file marks its registry entry as proposed and not yet integrated.

Methodology references

  • Method name(s): PlaceboTests (placebo-law / randomization-inference diagnostic; serial-correlation DiD inference)
  • Paper / source link(s): Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-in-Differences Estimates? QJE 119(1), 249-275. https://doi.org/10.1162/003355304772839588
  • Any intentional deviations from the source (and why): None — this PR adds a paper review only. The review faithfully distinguishes BDM-derived requirements from library extensions (leave_one_out, chosen-pre-period timing test, finite-draw p-value floor) and labels the diff-diff NaN-inference convention as a defensive enhancement, not a BDM requirement.

Validation

  • Tests added/updated: No test changes (paper-review documentation only).
  • Backtest / simulation / notebook evidence (if applicable): N/A — no executable behavior changed.

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown

Overall Assessment

✅ Looks good

No unmitigated P0/P1 findings. The PR is documentation-only and the added BDM review is explicitly marked as proposed, not an authoritative REGISTRY.md change.

Executive Summary

  • No estimator, weighting, SE, variance, default, or executable behavior changed.
  • The BDM scope is correctly framed as inference/diagnostics rather than a DiD estimator; the published abstract aligns with that framing. (doi.org)
  • The proposed PlaceboTests entry does not conflict with the current registry stub because it is clearly non-integrated.
  • Two P2 documentation/process issues should be cleaned up before relying on this as tracker-grade methodology evidence.

Methodology

  • Severity: P3-informational
    Impact: The proposed registry text is not yet authoritative. docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L28 says the entry is proposed, while docs/methodology/REGISTRY.md:L3098-L3107 remains the current edge-case-only PlaceboTests contract. This is acceptable for this PR.
    Concrete fix: No action required in this PR; validate and integrate into REGISTRY.md in the follow-up promotion pass.

Code Quality

No findings. No code changed.

Performance

No findings. No runtime paths changed.

Maintainability

  • Severity: P2
    Impact: The methodology tracker is now stale. METHODOLOGY_REVIEW.md:L1312-L1329 still says PlaceboTests has no canonical primary reference and only registry/tests documentation, and METHODOLOGY_REVIEW.md:L1464-L1466 still lists it as missing a paper review. That makes the new review hard for future source-validation work to discover.
    Concrete fix: Update the PlaceboTests tracker row/details to reference BDM (2004) and docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md, while keeping status In Progress if registry integration remains deferred.

Tech Debt

No untracked correctness debt introduced. The deferred registry/promotion work is called out in the new review at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L24; updating METHODOLOGY_REVIEW.md would make that tracking explicit.

Security

No findings. No secrets or executable surfaces added.

Documentation/Tests

  • Severity: P2
    Impact: docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L5 cites papers/119-1-249.pdf, but that file is not present in the repo. Other paper reviews usually state that local PDFs are gitignored and name the DOI/landing page as authoritative. This hurts reproducibility of the page/table checks.
    Concrete fix: Reword the PDF reviewed line to follow the project convention, e.g. published QJE PDF via DOI, local PDFs gitignored under /papers/, DOI version authoritative.

No test changes are needed for this docs-only PR.

@igerber @claude
docs(methodology): add Bertrand-Duflo-Mullainathan (2004) paper review
bcc4341 
PR-A of the PlaceboTests methodology promotion. Adds a paper-sourced review
of "How Much Should We Trust Differences-in-Differences Estimates?"
(QJE 119(1):249-275) grounding the diff-diff PlaceboTests diagnostic in the
placebo-law / randomization-inference framework, the serial-correlation
over-rejection problem, and the four inference corrections BDM evaluate.

Updates the METHODOLOGY_REVIEW.md PlaceboTests tracker row to reference
BDM (2004) and the new review (status stays In Progress; REGISTRY integration
and promotion remain for the source-validation pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@igerber igerber force-pushed the docs/placebo-bdm-2004-review branch from f50e3b4 to bcc4341 Compare June 26, 2026 18:51
@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: bcc434139fd46e34c1c6526cb54e1a468867f627

Overall Assessment

✅ Looks good

No unmitigated P0/P1 findings. This is documentation-only, and the two prior P2 review items are addressed.

Executive Summary

  • No estimator, weighting, variance/SE, inference, assumptions, or defaults changed.
  • The affected method surface is PlaceboTests only, as documentation/methodology tracking.
  • METHODOLOGY_REVIEW.md:L85 and METHODOLOGY_REVIEW.md:L1317-L1330 now reference BDM (2004) and track the remaining registry/test promotion work.
  • The new review clearly marks its richer ## PlaceboTests registry text as proposed, not authoritative, at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L28.
  • High-level BDM framing matches the DOI record: placebo laws, serial-correlation-driven SE understatement, and evaluated corrections via bootstrap, asymptotic VCV, and time aggregation. (doi.org)

Methodology

  • Severity: P3-informational
    Impact: The proposed PlaceboTests registry entry is intentionally not integrated yet. The current authoritative registry remains the edge-case stub at docs/methodology/REGISTRY.md:L3098-L3107, while the tracker explicitly leaves integration and a dedicated methodology test file as outstanding at METHODOLOGY_REVIEW.md:L1327-L1330. This is documented deferred work, not a methodology defect.
    Concrete fix: No action required in this PR; complete the listed follow-up by integrating the proposed entry into REGISTRY.md and adding tests/test_methodology_placebo.py.

Code Quality

No findings. No code changed.

Performance

No findings. No runtime paths changed.

Maintainability

No findings. The prior tracker staleness issue is resolved by the updates at METHODOLOGY_REVIEW.md:L1317-L1330 and METHODOLOGY_REVIEW.md:L1468.

Tech Debt

No untracked correctness debt introduced. Remaining PlaceboTests promotion work is explicitly tracked in METHODOLOGY_REVIEW.md:L1327-L1330.

Security

No findings. The PR adds methodology markdown only; no secrets or executable surfaces were introduced.

Documentation/Tests

No blocking findings. The prior PDF reproducibility issue is resolved at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L5, which now states the DOI/journal version is authoritative and local PDFs are gitignored.

Validation: git diff --check is clean. No tests were run because this PR changes documentation only.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Jun 26, 2026
@igerber igerber merged commit 606318c into main Jun 26, 2026
11 of 12 checks passed
@igerber igerber deleted the docs/placebo-bdm-2004-review branch June 26, 2026 19:15
@igerber igerber mentioned this pull request Jun 26, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@igerber